Approaches to Qualitative Comparative Analysis and good practices: A systematic review

The Qualitative Comparative Analysis (QCA) methodology has evolved remarkably in social science research. Simultaneously, the use of QCA too often lags behind methodological recommendations good practice. Improper is a serious obstacle for to enrich ‑methodology toolkit. We explore whether coherence analytic approaches can help us understand practices applied by performing systematic review 86 studies. Although adherence technical GPs improved over time, we find high prevalence incoherent, “hybrid” approaches. As hybridity study increases, its decreases. case-oriented, realist, exploratory studies do not consistently follow Instead, only good-practice approach but explicitly theory-evaluating. conclude that aligning choice with underlying and theory foster QCA. als sozialwissenschaftliche Methode hat sich bemerkenswertem Masse entwickelt und etabliert. Dennoch hinkt die angewandte QCA-Forschung den Empfehlungen zur Sicherung guter wissenschaftlicher Praxis oftmals hinterher. Unsachgemässe Anwendung stellt für ein ernsthaftes Hindernis dar, um existierenden Methoden der Sozialwissenschaft gewinnbringend zu ergänzen. Diese systematische Übersicht von QCA-Anwendungen untersucht, inwiefern Kohärenz des analytischen Ansatzes, dessen Rahmen angewandt wird, mit Einhaltung zusammenhängt. Obwohl technische verbessert hat, finden viele inkohärente, “hybride” Ansätze. Je hybrider eine Studie ist, desto mangelhafter ist Praxis. Der fallorientierte, realistische explorative QCA-Ansatz folgt nicht unbedingt konsistent Vielmehr erfüllt einzig realistische, theorietestende Ansatz zuverlässig diese Kriterien. Um verbessern, weitreichendere Integration Theorie hilfreich, analytische Ziele methodische Entscheidungen sollten besser aufeinander abgestimmt werden. L'approche (Qualitative Analysis) s'est développée remarquablement au sein sciences sociales. Pourtant, son utilisation est trop souvent en retard par rapport aux bonnes pratiques méthodologiques, ce qui constitue un sérieux à l'enrichissement de la boîte outils méthodologique A travers une revue systématique études basées sur l'approche nous examinons si cohérence approches analytiques peut aider comprendre les matière Bien que conformité techniques se soit améliorée fil du temps, constatons forte prévalence d'approches incohérentes et « hybrides ». Plus étude hybride », moins elle conforme avec pratiques. La seule approche constamment l'analyse centrée cas, réaliste, mais évaluant explicitement théorie. Les exploratoires, quant elles, ne suivent pas systématiquement recommandations. Nous concluons l'alignement choix analytique sous-jacente, d'une part, le recours théorie, d'autre peuvent encourager l'usage appliquée. This paper analyzes evolution link between published applications, based on review. qualitative (case-based) mixed-methods research combining different analytical methods yet as standardized quantitative tradition, it striving become ever more methodologically sophisticated, rigorous, transparent (Adcock & Collier, 2001; Brady 2010; Mahoney, 2010). Given relative novelty distinct logical foundations, method (Ragin, 1987/2014) – formalized set-theoretic comparative causal complexity, necessity sufficiency relations intermediate large sets cases offers highly interesting case these developments. undergone remarkable developments terms dissemination (Mello, 2021; Oana al., Rihoux 2013), innovation, diversification (Schneider, 2018; Thomann Maggetti, 2020). Methodological practice (GPs) using have been developed guide (Greckhamer Schneider Wagemann, Wagemann Schneider, 2015). Nevertheless, recent reviews state art suggest frequently (Emmenegger 2013; Møller Skaaning, 2019; Verweij Trell, 2016). In our opinion, this problem. Poor applications bear risk flawed inferences, also nourish criticism such. If toolkit sustainable manner, so through correct stringent researchers. However, lack proper understanding reasons low any usually question design coheres overarching goal given (King 1994). fundamental idea underlies distinction an technique truth tables minimization, one hand, encompassing approach, other (Berg-Schlosser 2009; Lobe, 2009). Unlike Creswell Poth (2016), define “the processes before after analysis data, such (re-)collection (re-)definition selection criteria, or (re-)specification concepts” (Rihoux 2015; 2012: 11). Ragin (1987, 2000, 2008) set out exploratory, integrative built foundation substantive theoretical knowledge 2021). challenging notion single QCA, recently Maggetti (2020) demonstrated including large-N no elements theory-evaluating (Møller 2019), focus mainly minimization algorithms (Baumgartner, partly contributed confusion QCA's use. argue each some choices are appropriate than others. While inherently superior another, they choosing tools line goals should improve assertion far gone untested. No exists about empirical types specificity their respective practices. performs applying public administration address three questions. First, which prevail interdisciplinary field, how prevalent “hybrids” mix approaches? Second, changed time? Third, linked particular not-so-good practices? bureaucracies political actors, nature (Rosenbloom, 1983) epistemological “affinity” toward core assumptions methodology, field typical contemporary indicative broader sciences.1 now elaborate Maggetti's typology then briefly introduce corresponding formulate expectations prevalence, development, relationship GPs. motivate outline systematically reviewing coding them. By mapping past current linking illuminate might get astray from pitfalls could be avoided. results indeed associated GP adherence. information insightful users, reviewers, teachers order move forward all areas What apart mainstream statistical complexity. conceives members non-members (Wagemann, 2017). condition necessary outcome if subset set. Conversely, sufficient Standard wisdom identifies so-called “INUS” causes; Insufficient Necessary parts configurations themselves Unnecessary, Sufficient (Mahoney Acosta, 2021).2 These resonate policy analysis, evaluation, democratization research, administration, amongst others Pattyn Rihoux, 2020; 2011; Thomann, Ege, 2019). Figure 1 highlights diversity currently being research: 1) cases, 2) mode reasoning, 3) explanation (Thomann Some apply coherently—the “hybrids”.3 Below specify illustrate hybrids defined. captures capitalize specific before, during, especially when drawing inferences. Rather sheer number observations, within-case what apart. case-oriented makes explicit individual during at least phases design, measurement, calibration, additionally uses (causal) interpretation results. strengthen inferences cross-case comparisons insights intimate familiarity (Rohlfing Rohlfing Zuber, reverse true condition-oriented does establishing focuses comparisons, conceptual robustness reliability solutions 2013). It combination ascertain validity (e.g., Meuer Rupietta, adopts hybrid neither draws token (within-case) level, nor type (cross-case) level. For example, via checks complementary analyses Or, limited interpreting Toots Lauri (2017) factors contributing endurance investment policies among six states Baltic region. guides steps, discuss findings light cases. Another example Alemán al. (2018) who perform 24 Columbian departments reveal multiple paths leading regional poverty. many decisions guided detailed country regions, cases.4 reasoning draw abstract knowledge. Traditionally, used generate rather test emerges evidence fashion. “help researcher new insights, may taken basis further development re-examination existing theories” 2009: 16). reflects iterative Thus, even inform calibration table analysis. prior observed.5 increasingly evaluating were derived notions Theory-evaluating differs traditional hypothesis testing deductive various respects. concrete set-theoretical confront results, instance formal set-analytic evaluation 1987; 2012). challenge epistemology. Expectations (combinations of) conditions consistent theory; claim hypotheses, hypotheses logic general directly testable. Shahidi (2015) explains patterns welfare recommodification aftermath economic crisis studying 18 labour market reforms advanced states. he very generally expects “[l]abour responses will vary function scope generosity arrangements” (Shahidi, 2015: 662). Using would possible disconfirm expectation. Marr (2012), investigates process exiting homelessness influenced interaction contexts operating levels. He formulates following hypothesis: “individual vulnerabilities acculturation less important pathways access resources” (Marr, 987). Such formulation vague evaluated Finally, diverge constitutes explanation. realist emphasizes interpretability6: just technique, drawn case, conceptual, derive meaningful entail empirically contradictory logically untenable claims. assesses relations, meaningfulness (Oana Asserting there causes necessary, therefore separates analyses. presence (when observed empirically), theoretically grounded counterfactual assumptions. procedure “intermediate solutions” 2008; “conservative solution” represents condensed complex description data patterns. More recently, divergent regularity causation (Mackie, 1965; models must redundancy-free idealist thus aims identify simultaneously outcome. diversity, solution reliably eliminates redundant “parsimonious solution”. parsimonious achieve coverage, indicating adopt protocol (i.e. enough coverage analyzing separately), (interpreting conservative though explanatory descriptive. instance, separately sufficiency, resort solution; incorrectly infer solution. Clearly, category cover variety choices. small-N discusses superficially, assess robustness. tend combine features incoherently, expect negative effect performance against phases, users commit mistakes. Accordingly, (2010: 398) recommended 2015, responding technological transparency initiatives, complemented emphasize need make well-informed reasoned decisions, establish highest levels them, connect interpret solutions. strongly Ragin's “traditional” vision qualitative, strong evolved, scholars continue value (Braumoeller, Cooper Glaesser, 2016; Thiem, 2016, Overall, refine original recommendations, without differing substantively them Rubinson Focusing earlier list (2010, 2015), analyze sample consider suggestions. (2021) updates explain detail implement them.7 rationale main steps (see e.g., Ragin, 2012; Kane Kahwati, Mello, Table lists indicates ones analysis.8 revolve around combinations conditions. distinguish related adopted. b) together project c) requirement moment d) There always justification (non-) e) kept moderate level f) selected conceptualized adequate well gained throughout 2 3 l) consistency research-specific, supported arguments p) negation dealt two separate j) Contradictory rows resolved minimizing algorithm k) Truth minimized computer software m) treatment (in csQCA) inconsistent fsQCA) n) remainders o) Based table, several formulas complexity produced presented 7 8 9 10 w) raw matrix x) reported y) Every contain formula(s) z) measures 11 12 13 14 q) Different presentational forms depict both case- variable-oriented aspects r) back mechanical way s) Solution preferably graphical representation 15 16 t) Individual conjunctural equifinal term (over)interpreted u) provide justifications (or more) towards deemed v) formula alone demonstrating 17 Study 2015 ) Were unions passed threshold reported? 19 20 phase (GPs 1–4) entails definition conditions, minimize problems diversity. GP3 suggestion share uncovered rows.9 During step defined attributed sets. calibrated sets, analysed parameters fit evaluate 5–10). measure relation data. importance relations. GP6 foresees performed from, to, sufficiency. begins attributing determining, row, indicated score. Through comparing detecting software-based shortest expression those imply outcome—the formula. process, researchers decide wish unobserved (logical remainders). types: conservative, intermediate, GP10 however applicable under explanation, deems causally interpretable. After 11–20), diagnostics tell robust Since analytically interested cannot relate (GP15 GP16 applicable). Under tests desirable unambiguous (GP19 presentation 1987/2014). review, include standards operationalized expected across approaches, closely standards. justify exclusion certain (not marked bold box A1 Online Appendix. leaves total Each article's assessed calculating met (applicable) Yet, since software-related innovations time provided opportunities checks, tools, calculate fit, most subject applicability criterion A4). GP, was publication10 relates article conduct Moreover, already discussed, meaningfully ask prevail, give rise shortcomings Box summarizes hypotheses. H1: type. Results: No, containing frequent. Both regarding frequent pure common approach. H2: relatively rare. Yes, 3.5 percent show . H3: Condition-oriented phenomenon 2013 H4: Compliance particularly moment. QCAs substantially improvements strongest significant H5: Hybrid prone failing Looking mixed evidence. types, reasoning. clear that, however, quite fail meet H6: best. comply high-performing put prominent textbooks predominantly taught schools. popular management business early 2010’s Similarly, spotlighted Baumgartner 2020) represented major time. 2010, actively disseminated teaching peer particular, referring complied fully years, reflecting rapid ease implementation replicability (Dusa, 2018). Our “any select remain faithful them” 2020: 21). King al.’s (1994: 7, 12) call “disciplined thought”: procedures inquiry inferential rules. matching decisive. approac